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Abstract 

Information Visualization techniques are built on a context with many factors related 
to both vision and cognition, making it difficult to draw a clear picture of how data 
visually turns into comprehension. In the intent of promoting a better picture, here, 
we survey concepts on vision, cognition, and Information Visualization organized in a 
theorization named Visual Expression Process. Our theorization organizes the basis of 
visualization techniques with a reduced level of complexity; still, it is complete enough 
to foster discussions related to design and analytical tasks. Our work introduces the fol¬ 
lowing contributions; (1) a Theoretical compilation of vision, cognition, and Information 
Visualization; (2) Discussions supported by vast literature; and (3) Reflections on visual- 
cognitive aspects concerning use and design. We expect our contributions will provide 
further clarification about how users and designers think about InfoVis, leveraging the 
potential of systems and techniques. 
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1 Introduction 


Understanding why and how visual representations work is an important issue addressed in 
many works on Information Visualization ( Dill et al.|2012 )( Thomas fc Cook[2005 ). On the track 
of this issue, in this work, we assume that well-designed visualizations must stimulate visual 
and cognitive processes in a way that reasoning is amplihed. The effective promotion of such 
reasoning depends on principles mastered in the sciences of vision and cognition: vision is the 
gate through which information derived from computer graphics reaches the brain; cognition 
refers to the processing that is induced by such graphics. Vision and cognition are closely 
intertwined, a fact to be considered in the design of visualizations. Accordingly, understanding 
Information Visualization (InfoVis for short) in light of these sciences may improve design 
principles that, usually, are performed intuitively. To this end, we review the steps that take 
place during vision-cognition phenomena when the goal is data analysis. In survey fashion, we 
compile the literature introducing the following contributions: 


• Theoretical compendium: we draw a descriptive relationship between vision, cognition, 
and Information Visualization; 

• Discussions: we debate our rationalizations over extensive literature; 

• Reflections: we provide study cases to revisit design practices. 


We organize visualization concepts aiming at the needs pointed out by Johnson et ah (2006), 


who recommend the characterization of how and why visualizations work, and by Scaife fc 


Rogers (1996), who stress the importance of the cognitive aspects underlying visualizations. 


Furthermore, we review principles for visual representations that, according to Card et ah 


(1999), are an initial step towards more effective visualization techniques, a demand defended 


by |Wong et al.| ( p012[ ). 

We note that, while we build an association between Information Visualization, vision, and 
cognitive sciences, we do not reach a definitive settlement. This is because neither vision nor 
cognition are yet fully understood. Rather, constrained to the current state of the art, we 
introduce an organizational process that discusses use and design. 
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2 Related work 


The literature presents several models that lay the basis to Information Visualization. Bert in 


(1977/1981) introduced the concept of deriving visual structures from a set of fundamental 


components. Cleveland & McGill (1984), and Mackinlay (1986), conducted studies on the use¬ 


fulness of visual patterns in the form of frameworks for design. Card et ah (1999) follow Berlin 


by discussing the importance of the spatial substrate. Keim (2002) suggests a taxonomical 


space for quick referencing; Shneiderman (1996) reasons about the possibilities of visualization 


and interaction; and Chi|(2000) focuses on data transformations. In the realm of design, Buga- 


jska (2003) deals with spatial and abstract visualizations considering the design guidelines of 


Tweedie (1997). In the line of works that reflect about the visualization held, van Wijk (2006) 


systematically discusses visualizations based on a cost-oriented analysis; and Green et al. (2009) 


research many cognitive and perceptual aspects (Rensink 2000) to build a model and a set of 


guidelines for design. Patterson et al. (2014) introduces a framework based on vision and cog¬ 
nition sciences, similar to ours, focusing on top-down processes. Comparatively, we conduct a 
complementary bottom-up approach; we translate vision-cognition phenomena into an original 
vocabulary that may bring such science closer to the design practice. 

To organize our compilation on vision and cognitive sciences, we depart from the Visualiza¬ 


tion Pipeline of Card et al. (1999) to draw the Visual Expression Process, a sequence of events 
delineated by the possibilities of the visual-cognitive interplay. According to our organization, 
(1) vision phenomena (pre-attentive stimuli) determines a map of potential interesting objects. 
Then, attentive selection concentrates on one single element, part of a set of (2) analytical 
perceptions. Such perceptions go through a pattern-matching process that turns them into 
(3) abstract patterns that, in working memory, support cognition in combination with domain 
knowledge originated from long-term memory. Finally, leading to (4) cognitive decision sup¬ 
port. Nevertheless, our process does not explain the intercourse between vision, cognition, and 
Information Visualization; this is not feasible considering the current knowledge, and neither 
it would ht in a single article. 

The rest of the paper is organized as follows. Section draws a detailed panorama of 
vision and cognition in the realm of Information Visualization. Section introduces the Visual 
Expression Process, our organizational theorization; and the last section presents conclusive 
remarks. 
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3 Concepts on Vision, Cognition, and Visualization 


According to Vision Science, the visnal process has two stages, namely, the parallel extraction 
of low-level properties, called pre-attentive processing, followed by a slower detailed scan. The 
hrst stage promotes the major beneht of visualizations, that is, improved data comprehension 


(Triesman 1985). Meanwhile, the second stage addresses conventional reading practices that 
do not contribute towards faster cognition, but that are necessary for further analysis. In fact. 


Ware (2004) states that understanding what is processed pre-attentively is probably the most 


important contribution that Vision Science can bring to visualization. 

This two-stage process might be the underlying motivation for the broadly referenced “visu¬ 
alization mantra”: overview hrst, zoom and hlter, then details on demand ( Shneiderman||1996 ) 
- Schneiderman came to this conclusion by means of intuition and empirical experiments, never 
drawing the vision/cognitive reasons of why this is the case. We discuss this process in the 
following sections according to our process, presented in section 


3.1 Maps of Saliences 


In the early stages of vision, the brain deals with the problem of casting potential elements of 
interest; regions of the scene that should be considered for cognition. To cope with that, a com¬ 
plex process takes place so that certain characteristics pop out to the eyes. These characteristics 
appear as saliences over the scene. They dehne the so-called maps of saliences, exemplihed in 
Figure [T] the hrst element of our theoretical organization. 



Figure 1: (a) Example of a map of saliences over the Gapminder tool. In the scene, the bubbles 
with outstanding features stand for saliences corresponding to countries of interest, (b) The 
result is a map of potential targets that pop out due to their color, or their shape. 


The principle of salience is to reinforce the perception of the areas in the scene whose visual 
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properties contrast with those of their surroundings (Itti & Koch 2001). It is a process that 


considers different visual aspects, such as position and color, and that ends as it combines them 


onto a single scene (Nothdurft 2000) 


The occurrence of salient features stems from their interplay with other stimuli, depending 
on a context that favors conspicuousness. The brain is highly trained to detect such configura¬ 
tions, being able to track salient features in parallel, in real-time, and covering the entire visual 
field. There is also evidence that this ability is strongly influenced by the task-at-hand, in a 


top-down (cognition-to-vision) process (Navalpakkam & Itti 2007) (Patterson et ah 2014) 


3.2 Attention and Attentive Selection 

Although the brain perceives visual targets simultaneously in a map of saliences, it cannot 
process all of them in parallel. This is considered a prohibitively computational task even 


to the most sophisticated brains (Tsotsos 1991). Primates and other animals handle this by 


restricting the consideration of the objects presented to their eyes: their vision concentrates 
on small regions considering objects one after the other. This is a serialization process ruled 
by what is called attention. In other words, once a map of candidates is ready, it is necessary 
to “filter out” one of these candidates for attention. Attention, here, occurs in accordance 
with the task that the user is performing; the task determines the information demands and, 
consequently, what visual stimuli should be extracted from the scene. Trial and error is part of 
the process, which is iterative. 

This process of filtering has been modeled as a pyramidal neuronal structure - named 


selective tuning model (Cutzu & Tsotsos 2003) (Essen et ah 1992). This theoretical model 


predicts a broad layer of neurons in its first level, narrowing down as it advances to upper layers. 
The layers intercommunicate through feed-forward and feedback connections and, according to 
Cutzu & Tsotsos (2003) ( Tsotsos et al.||1995[ ), a pyramid of neurons successively performs three 
stages of processing, illustrated in Figure (a) bottom-up feed-forward, (b) top-down winner- 
take-all feedback (Lippmann 1987[ ), and (c) bottom-up straight path. This process explains 
what is broadly referenced as filtering or attentive selection. 

Following this widely-accepted model, it is worthy to note that, as described, filtering occurs 
according to both bottom-up and top-down processes. The bottom-up process depends on the 


visual stimuli, while the top-down process depends on the task at hand and is prioritized (Wolfe 


1994). Each process influences the other iteratively. Actually, according to Patterson (2012), 


the entire visual-cognition interplay in influenced by top-down mechanisms that emanate from 
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working memory, but that may be highly influenced by long-term memory (Woodman et ah 


2013). In this work, we focus on the bottom-up process, but there are works that explore 


higher-level factors related to the top-down process (|Patterson et ah 2014). 
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Figure 2: Three-stage pyramidal visual selection: (a) bottom-up feed-forward, (b) top-down 
winner-take-all feedback dLippmaim 1987), and (c) bottom-up straight path. 


3.3 Cognition, Memory and Vision 

After a target is selected, it is potentially useful for “details on demand”, or cognition. In 


general terms, cognition refers to the acquisition or use of knowledge Cohen| (1985), and may 
occur analytically (conscious and slower) or intuitively (automatic and fast), as defended by the 
dual system theory (Evans 2008[ )( Evans fc Stanovich||2013 ). Cognition is of special importance 
in data visualization as it supports analogical reasoning (Patterson et al. 2009). That is, 
the transfer of inferences from a relationship of elements in one domain (the analogue) to a 
relationship of elements in another domain (the target). In any case, cognition is intermediated 
by the memory system. The relationship between memory and cognition is studied by works 


on Cognitive Architectures, such as ACT-R (Buttner 2010) and Soar (Young & Lewis 1999). 
In Soar and other theories, the structural configuration of memory roughly reflects the model 


of Baddeley & Hitch (1974). Working memory includes three components: the central executive 
module, the phonological loop, and the sketchpad. The central executive module determines 
the attention focus, guiding the visual system, for example, by top-down biasing the pyramidal 
selection mechanism. The phonological loop stores information related to sound. The sketchpad 
(also known as Visual Short-Term Memory - VSTM[^ is associated with the maps of saliences 


^although, this is not a consensus 
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discussed in Section |3.1[ storing information related to space and to visual features. 

Following these lines, memory comes to be the main element in supporting cognition as it 
allows complex mental operations. Consequently, it also supports the practice of data visual¬ 


ization. Miller’s Law (Miller 1956) states that the memory is limited to 7-I-/-2 elements; more 
recent, and accepted, works state that this limit is at 4 chunks of elements - a chunk refers 


to the grouping of elements into larger units based on their meaning (Cowan 2010). Alvarez 


& Cavanagh (2004) discuss memory limits considering the nature of the elements to be re¬ 


membered, demonstrating that, despite the consensus of a (very) limited resource, there is not 
an ultimate conclusion about the topic. In any case, these limitations are severe because the 
greater the capacity of an individual’s memory, the more information she/he has available for 


solving problems (Just & Carpenter 1992). 

The memory system supports cognition in two ways: by retaining a list of elements for 


quick referencing, and by assisting in the construction of mental models (Johnson-Laird 1983). 


According to Johnson-Laird (2010), mental models preserve the relationship between entities 
by dehning analogies that save on logical reasoning, one of the principles behind complex 
visualization techniques and interaction (Liu fc Stasko|^10a|. Following the study of Logie 


(1995), mental models are created in the visuo-spatial sketchpad, a specialization of VSTM. 
Stimuli encoded into VSTM, ruled by cognition, may activate enduring information known as 


long-term memory, including facts, meanings, relationships, skills, and procedures (Patterson 


et al. 2010) - for a deeper discussion on working memory and long-term memory, refer to the 


work of Rose et al. (2010). 

The concepts presented in the former sections rely on ideas posed by widely accepted the¬ 
ories, among several others, for the visual-cognitive process. The choice for this specihc line of 
thought has been motivated by its intuitive coherence and scope of influence in the literature. 


Notwithstanding, other theories are widely referenced, such as the Spotlight (Eriksen & Hoff¬ 


man 


1973) and the Gradient (Cheal et al. 1994) models. For an ample discussion, refer to the 


work of Squire (2009). 


4 The Visual Expression Process 

In this section, we review the practice of visualizing data by considering the concepts presented 
so far. We organize the relationship among visualization, vision, and cognition according to 
a framework named Visual Expression Process - Figure Our theoretical organization has 
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four constituents: (1) pre-attentive stimuli, (2) analytical perceptions, (3) abstract patterns, 
and (4) decision support. For completeness of our survey. Table summarizes previous models 
found in the literature. Our theorization is inspired by these previous works putting together 
concepts in a complementary point of view. 

Table 1: Summary of previous models on InfoVis. 


Model 

Summary 

Lohse(1993) 

An algebraic model to estimate the effort to answer 
questions based on a visualization. 

ICard et al.l 

A pipeline of how to conduct the visualization 

(1999) 

practice. 

van Wijk (2005) 

An economic model stated to evaluate efficacy and 
efficiency. 

(Ware||2005) 

A top-down (problem-solving) model that states that 
what we see in a visualization depends on what we 
are seeking for. 

(Hegarty||2011 1 

A descriptive discussion of how visualizations work 
that culminates into a compilation of principles and 
perspectives. 

(Patterson et al. 

A vision-cognition model that explains visualization 

2014) 

and how to improve it from a top-down perspective. 


According to our theorization, (1) pre-attentive stimuli come from the neuronal reaction 
to light, determining a map of potential interesting objects, or saliences. Then, attention 
concentrates on one single element, part of a limited taxonomic vocabulary of (2) analytical 
perceptions. Such perceptions go through a pattern-matching process that turns them into 
(3) abstract patterns. Finally, abstract patterns in working memory support cognition in 
combination with domain knowledge originated from long-term memory, leading to (4) decision 
support. Notice, in the figure, that the arrow in between each pair of the four constituents is 
bidirectional to indicate that the interplay occurs both bottom-up and top-down, notably in 
response to the task at hand. Also, notice that after cognition, there might be new information 
demands, what leads to the redehnition of the visualization by means of new parameters for 
interaction - as depicted in blue in Figure Following, we present further details. 
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Figure 3: The Visual Expression Process for Information Visualization specified with four 
constituents. The black bidirectional arrows refer to bottom-up/top-down visual-cognitive pro¬ 
cesses. In blue, new parameters of interaction alter the scene iteratively. 


4.1 Pre-attentive Stimuli - channels for data encoding 

Pre-attentive stimuli impel maps of saliences, as highlighted on the leftmost side of Figure 
In the work of Rodrigues et al. (2008), the authors verified that such stimuli manifest through 
position, shape, color, and time. Here, we refer to these factors as channels for data encoding, 
in the sense that they encode data into visual stimuli. 

Although the consideration of four channels is a reductionist classihcation, it is supported 


by the literature. About color and texture, Watt (1995) affirms that, just like texture, color 
is the psychological response to the spectral characteristics of a surface; and that, different 


surfaces are perceived as having different colors. Furthermore, Motter (1994) observes that, 
early in visual processing, the incoming information is sorted and grouped according to the 
similarity of simple shape features, such as orientation or size, and of surface features, such as 
color, luminance, or texture. 


The features of each channel span to a large set, but Card et al. (1999) observe that just a 
limited number of the many existing graphical properties are used for Information Visualization. 


Healey & Enns (2012) present an extensive survey on pre-attentive features and visualization. 


A non-exhaustive list of such features is presented in Table 
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Table 2: Classes/channels of pre-attentive features from the perspective of data representation. 


Pre-attentive 

class 

Features 

(data encoding channel) 


Position 


1D/2D/3D position, stereoscopic depth; 

Shape 


line, area, volume, form, orientation, length, width, 
collinearity, size, curvature, marks, numerosity, con¬ 
vex/concave; 

Color 


hue, saturation, brightness and texture; 

Time 


movement, morphing, blinking, color/light intermit- 



tence. 


Discussion 


A well-designed visual representation must present a high overlap between its map of saliences 
and its (implicit) map of semantic relevance. That is, the design of visual representations is 
supposed to maximize pre-attentive effects. However, such maximization may not be possible 
without flexible human intervention over the channels of data encoding. In other words, inter¬ 
action, the active redehnition of the channels, is a mandatory need not fully explored in many 
designs. 


Liu et ah (2008) point out that many visualization systems are not sufficiently flexible to 


support user customization and appropriation. In fact, it is not difficult to find visualization 
tools that are limited in allowing the user to determine how to encode data. In these circum¬ 
stances, a user may ask “may I change the positioning order of the elements?”, “can I have 
each year represented with a different shape?”, “can I color the left group in red?”, or “can I 
see that animated?”. Each of these examples refers to a particular pre-attentive feature or, as 
we propose, to a data encoding channel. 


For instance, consider the seminal system GGobi (Gook & Swayne 2007), which introduced 


a large set of features if compared to its former version, system XGobi. Still, a brief analysis 
reveals that there is much to be improved: positioning of views is limited, except for Parallel 
Goordinates; shape is not an option for coding in the same way that color is; and animation 
is restricted to scatter plots through touring techniques. Those design issues contrast to what 
is observed in commercial systems like TIBGO’s Spotfire (http://spotfire.tibco.com/) and 
Google’s Gapminder (http://www.gapminder.org/), which present higher levels of freedom 
for each encoding channel, but that, still, do not achieve full appropriation of the scene the way 
we discuss. 

Accordingly, the design of visualizations must not only allow users to alter visual attributes. 
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they must incorporate mechanisms to emphasize each channel. This is necessary because a given 
stimulus (position, shape, color, or time) may fail to capture attention when it is surrounded by 
other stimuli - this is called inattentional blindness. In such circumstance, this given stimulus 
fails in competing with other more interesting stimuli or, worse, it is ignored (Mack A]|1998)(DJ 


2000). “Blindness” occurs especially when the viewer has some previous expectation on the 


scene; in such situations, top-down processes can strongly influence on what is noticed. This 


issue has been noticed by Patterson et ah (2014), who claim that the design of a visualization 


technique must provide means to capture attention alerting users of changes. 


Furthermore, according to the limitations of the memory system, as presented in Section 3.3 


memory overloading is a problem that might constrain problem-solving. In fact, according to 


Wickens (2008), multiple tasks in one same cognitive dimension might decrease performance. 


One possibility to lessen this drawback is to selectively turn the visual channels on and off. 
For example, it is possible to conceive a dispersion plot in which size is simply turned off, 
so to temporarily avoid overlapping - see Figure (b). Similarly, it is possible to turn off 
color, presenting all the information with black marks, emphasizing the role shape contours 

- Figure l^c). It is also possible to include intermittence, or movement, much the way that 
cyclists do with lights in urban traffic. This way, specihc graphical elements can start to blink 

- Figure |^e). We conjecture that this kind of selective constraining of channels not only 
alters attention mechanisms, as there will be fewer candidates for attention, it also reduces the 
memory load, as there would be fewer targets. Furthermore, it has the potential of affecting top- 
down processes by de-emphasizing features that are expected by the user - and that prevent 
her/him from noticing unexpected traits due to “blindness”. In these cases, algorithms to 


assist visualizations may be helpful, as advocated by Chen (2006). Such algorithms might 


monitor statistical features or mine characteristics of interest to suggest new encodings “on the 
fly”, enabling more intense appropriation of the scene. However, not much research has been 
conducted on this issue, which has potential for further developments. 


4.2 Analytical Perceptions 

Data encoding channels provide maps of potential targets for attention. Now, following vision 
theory, the next mechanism is attentive selection - see Figure Biased by user intention, a 
subset of the prominent entities in a visualization will reach working memory. Once selected, 
the chosen visual stimuli will be the basis of the analogies that lead to mental models, see 


Section 3.3 Here, one question comes up - what perceptions are produced by the targets of 
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simultaneously 
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Figure 4: (a) Dispersion plot using channels of position, shape contour, shape size, and color, 
(b) Shape size turned off. (c) Color turned off. (d) Shape contour turned off. (e) Time 
movement turned on for automatically detected outliers, (f) Shape size turned on. 


attention in an information visualization design? 

To answer this question, we have extensively inspected the literature tracking the ways in 
which visual manifestation occurs when the intent is data analysis. We have found a limited set 
of possibilities, dehning a visual taxonomic vocabulary whose elements appear recurrently. We 
refer to these elements analytical perceptions, depicted in the second part of Figure In the 


realm of interaction, Yi et ah (2007) followed a similar procedure based on extensive literature 
inspection; they achieved results concerning the user intent, producing a limited set of recurrent 
possibilities. 

Analytical perceptions are the traits that any user attentively seeks for in a visual repre¬ 


sentation. Following the dual system theory presented in Section |3.3| , such perceptions occur 
intuitively. Our investigation indicates that such elements include correspondence, differentia¬ 
tion, recognition, connectivity, arrangement, and variation in time. The most verihed of these 


phenomena, correspondence and differentiation, are noted by Bertin| (1977/1981) and by Card 


et ah (1999). The third analytical perception is presented by Mackinlay (1986) who states 


that the notion of relationship among graphical entities comes from the perception of connec¬ 
tivity. Meanwhile, arrangement arises from group positional conhgurations, largely studied by 
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the Gestalt psychology (Koflkap.935) as observed, for example, in graph layouts (|Dwyer et ah 


2009) pbllis et ah 1998). Recognition^ in turn, takes place as a resemblance to previous knowl¬ 


edge and/or expertise, a concept studied in psychological models and Information Visualization 


models. As Liu & Stasko (20106) point out, the concept of internalization involves the encod¬ 
ing of perceived information into long-term memory (enduring information or pattern). Lastly, 
variation, manifests only along time - not necessarily for temporal data - and in combination 
to the other hve perceptions. 

The notion of analytical perceptions becomes evident when they are not found and the 
pipeline outlined in Figure is broken, preventing Visual Expression - if none of the aforemen¬ 
tioned perceptions occur, the user is unable to make sense. As depicted in our theorization, 
analytical perceptions occur after pre-attention (Section [4.1[ ) and before abstract patterns (Sec¬ 
tion 4.3), independently of the data domain. Hence, they bridge vision and data interpretation. 


Specihcally, we discuss the analytical perceptions and how they relate to the data encoding 
channels in the following. 


correspondence-, each position/shape/color has a direct correspondence to a referential 
map - discrete or continuous - that is part of the scene (explicit) or that is mental (im¬ 


plicit), dehning analogical reasoning as explained in Section 3.3 Explicit maps include 
axes, geographical maps, shape/color dictionaries, and position/shape/color ranges. Im¬ 
plicit maps include known orderings and shape metaphors; 


• differentiation: each position/shape/color discriminates graphical items. Differentiation 
is a correspondence achieved by the user, who creates a referential map in memory. Such 
map is limited in the number of elements (or differentiations) according to the limitations 
of memory; 


• recognition: positions/shapes/colors whose decoding comes from the expertise of the user 
or from previous knowledge - recognition is a correspondence established from visual 
entities to concepts retained/learned in long-term memory; 


• connectivity: shapes, mainly edges, that convey information about relationships among 
entities in memory; 


• arrangement: Gestalt principles of organization - positional placements (closure, proxim¬ 
ity, and symmetry) that convey perception about group properties, for example, clusters 
and structural cues; 
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variation in time: obtained when the parameters of position/shape/color are altered along 
time, indncing new perceptions for each of these channels. 


Discussion 

Onr set of analytical perceptions is not an exhanstive listing, bnt a hrst reference that snggests 
that visnalization designs tend to resort to the same basic set. For instance, the design of 
Google’s Gapminder tool, althongh contemporaneous, reproduces the same dispersion plots of 


statistical books a hundred years old. Nevertheless, as observed by Liu et ah (2014), works on 


new designs and on the evaluation of existing ones have not considered that there is a limited 
set of elements to instantiate in data representations. This fact could fruitfully support the def¬ 
inition of design languages and frameworks, which would beneht from recurrent constructs that 
lead to a limited set of analytical perceptions. Differently, current languages and frameworks 
rely on graphical patterns and it is up to the user to build the desired analytical perception; 
see Table for a representative set. 


Table 3: Previous works on languages and frameworks for visualization design. 


Work 

Approach 

Elements 

Protovis 

(Bostock & 

Heer 2009)(Heer 
& Bostock||2010 ) 

Graphical 

Marks, shapes and layout 

(Bostock 

Visualization 

Selection, operation, join, layout and 

et al.||2011) 

pipeline 

transformation 

Improvise 
( Weaver||2004 ) 

Link and 

coordinate 

Variable, function and view 

Prefuse (Heer 

et al.|2005 ) 

High-level API 

Filter, layout, interaction, color and 

size 

ggplot2 
(Wickham 

2009) 

Domain specific 

Layer, scale, coordinate system and 

facet 

Flexible Linked 

Axes (Claessen 
& van Wijk| 
2011) 

Linked axes 

Axes mapping, interaction, line and 
point 

This work 

Visual-cognitive 

Pre-attentive encoding channels and 
analytical perceptions 


By considering the concepts reviewed so far, it is possible to conceive a design language 
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whose approach is based on cognition, and whose elements are interactive encoding channels, 
and analytical perceptions - refer to Table for comparison. For example, in this design 
language, one would be able to state a visualization by choosing channel co/or and perception of 
differentiation. This same visualization would demand a few more elements, as channel position 
and perception of correspondence. As in any language, these elements would receive parameters 
according to an extensible library of data-to-marks mapping. In the case of relationships, 
one would be able to choose channel position with perception of arrangement, together with 
channel shape and perception of connectivity. Underlying algorithms would abstract undesired 
complexity, as an algorithm for outlier detection to support color differentiation; and a force- 
directed algorithm to support arrangement. As a design language, this approach would bring 
the beneht of discriminating the recurring elements of visualization techniques and having 
them in libraries for composition. This alternative approach contrasts with the usual practice 
of combining recurrent elements in ensembles assumed as new techniques. 


4.3 Abstract Patterns 


According to Hutchins (1996), tools - or externalizations (Hegarty 2004) - transform difficult 


tasks into in-mind manipulations of physical systems, or into pattern-matching problems (Giere 


2002). Pattern matching, or pattern-recognition (|Patterson et ah 2014), the association of a 


given stimulus to information retrieved from memory (Eysenck & Keane 2003), is one of the 


principles of data visualization and the second step of the Visual Expression Process - see 
Figure 

According to our organization, once a user focuses on an analytical perception, she/he 
proceeds to match this perception to an abstract pattern. Based on the theory of vision - seen 


in Section |3.3[ the generation of patterns is supported by memory, which is hlled with data 
from the visual-sensorial system or from long-term memory. The efficiency of this intercourse is 
explained by the fact that the visual-sensorial system provides spatial information to memory 


at rates higher than that of long-term memory (Ware 2004), in a time ranging from 100 to 


250 ms (Kieras & Meyer 1997). Hence, analytical perceptions in the visual-sensorial system 


work similarly to the images stored in long-term memory, providing efficient pattern-matching. 


According to the dual system theory. Section |3.3[ this process is conscious and slow; but, 
with practice and experience, a given analyst can become a specialist. For specialists, pattern 


matching becomes intuitive (Evans 2008) - refer to the work of Patterson et ah (2010) for a 


thorough discussion. It is a straight conclusion, then, that a period of practice and experience 
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is necessary for most visualization techniques, and that this is an obstacle for use. 

A suggestive set of the patterns - third part of Figure |^- that arise from the perception-to- 
pattern matching, includes: correlation, tendency, classihcation, relationship, order, summa¬ 


rization, outlier, cluster, structure, and reading. Tufte (1997) and Amar et al. (2005) provide 


more exhaustive listings that follow different rationalizations. 


Discussion 

In our literature review, we noticed that many systems disregard the fact that visualization 
techniques converge through abstract patterns. According to current practices, instead of a 
set of familiar patterns, the user has to choose among a set of visualization designs that. 


quite often, they have little experience with (Chen 2005). Thomas & Cook (2005) reinforce 


this notion; they state that abstract patterns correspond to the second factor of their four- 
steps analytical-reasoning process. Following their process - a pattern-to-construct sequence, 
users have constraints in relation to what they can search for in face of a given pattern. For 
example, suspiciousness tends to appear by means of tracking for outliers; while evidence of 
illegal lobbying practices emerge from clusters; and community detection in graphs is a task for 
relationship. Still, users are often offered a menu whose options are, for instance, dimensional 


stacking (LeBlanc et al. 1990), star coordinates (Kandogan 2000), and table lens (Rao & Card 


1994); alternatives far from the pattern-to-construct task they have in their minds. A similar 


notion dates back to the 1990’s, in the work of Wehrend & Lewis (1990), who advocates in favor 


of more specihc problem-oriented choices. We suspect that, because design-oriented interfaces 
neglect the more natural notion of abstract patterns, this is possibly one of the reasons why 
advanced visualization techniques have struggled to achieve a wider commercial dissemination, 
e.g, in office suites and in everyday spreadsheets. 


Take, for instance, the visualization technique Treemap (Shneiderman pd)92), introduced 


for visualizing hierarchical data in general. In two decades, Treemap gained popularity at 
the academy, but, as a general hierarchical tool, it has failed in reaching a wider use. This is, 
possibly, because it has been criticized since its introduction ( Barlow &: Neville|2001 , Cawthon & 
Moere 2007| Fabrikant & Skupin 2005), being accused of lacking cognitive plausibility, having 


poorly perceived aesthetic qualities, and presenting poor task-driven performance (Wood & 


Dykes|j20'08 ). Despite all, however, an especial design of the Treemap has remarkably succeeded. 
The SequoiaView system (www.win.tue.nl/sequoiaview) has achieved wide dissemination 


(check (van Wijk 2006) for some impressive numbers), far beyond the academic walls. But, 
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how can we have two flavors of the same technique evolve in different ways? Certainly, not 
one single aspect explains everything, but an outstanding factor comes up: the SequoiaView 
is domain and pattern-oriented, it is distributed to visualize the structure and the sizes of the 
files in your hard-drive, specifically. In accordance, users do not have to discover that the tool 
is good at doing this; instead, when they have this specific problem at hand, they are guided 
to SequoiaView, a more natural process. 

In designing systems, an alternative course of action would be to initially present the user 
with a set of patterns to choose from. After that, she/he would be offered a set of visualization 
techniques that better suit the pattern they are seeking for. Users may know what to look for 
by means of previous knowledge of the data domain, by means of known problems to be solved, 
by means of suspicious clues perceived along the data usage, and also by means of previous 
visual exploration of the data. 


4.4 Cognitive Decision Support 

After producing patterns, vision is no longer an active agent, neither pre-attentively nor atten¬ 


tively. Now, the analysis follows the widely accepted pattern-then-cognition process (Margolis 


1990) (MacEachren 2004) to achieve decision support. Indeed, according to the analytical rea¬ 


soning of Thomas & Cook (2005), Information Visualization cannot ultimately provide decision 
support, which can only be achieved after interpreting the patterns in light of the data domain 
- rightmost side of Figure That is, even though users can come up with patterns without 
considering the underlying data, these patterns are not of great use if the domain is not deeply 
understood. 

Insufficient domain knowledge leads to unsatisfied expectations in relation to InfoVis, cre¬ 
ating situations in which a user is presented to a supposedly insightful visualization but, then, 
everything one hears is “so what”? A disappointment that happens due to the enthusiasm 
according to which one can solve a wide range of problems just by looking at the data. How¬ 
ever, visualization tools can do little if the analyst is not well-prepared to assess what the data 
ultimately describes and potentially carries within. 


Discussion 

To prevent the aforementioned problems, InfoVis systems should define systematic means to 
aid the user in recording and accessing the domain knowledge related to the problems at hand. 
An interesting approach to overcome the gap of domain knowledge is to use annotations (or 


17 










analytic provenance (Xu et ar]|2015)), either automatic or manual. As pointed out by Hullman 


et al. (2011), annotations help to direct the user’s attention and foreground particular insights, 


supporting the most efficient inferences. The work of Hullman et al. (2013) exemplihes this 
issue. Their work focuses on stock-price time series, which are hard to understand if one is 
not aware of the facts that influenced the behavior of the market. Their system solves this 
problem by identifying news that happened contemporaneously to outstanding patterns found 
in the plots, presenting them on demand - illustrated in Figure]^ Similar approaches (Dennis 


et al.||2003 Conesa et al.||2005) have been proposed for genomic data, an extreme case in which 


domain knowledge is necessary, otherwise no reading of the data (either visual or textual) will 
make sense. 
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Figure 5: Example of an annotated visualization produced with Contextiher - in the figure, 
labels provide domain knowledge. Reproduced from the work of Hullman et al. (2011|[). 


An alternative is to use technique storytelling, as surveyed by Segel & Heer (2010). Plaisant 


(2005) defends that advanced interfaces need to address the long-term process of analysis that 
may require annotation, history keeping, collaboration with peers, and the dissemination of 
results and procedures used. Storytelling not only attacks the problem of lacking domain 
knowledge, it also provides knowledge about new hndings in the form of further “story chapters” 
interactively created. The visual analysis, potentially, becomes an incremental set of insights 
from multiple experts in the form of bookmarks, keyword tags, text comments, and audio 
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annotations. 


By bringing existing (factnal) or prodnced (annotated) domain knowledge to the stage, 


abstract patterns can snpport reasoning by snpporting conditional inferences (Johnson-Laird 


& Byrne 

2002 

), possibly influenced by statistical regularities ( 

Patterson et ah 

2012 


The 


resnlt is the provision of inferences for decision making, notably, predictions, alternatives, and 
evalnations as depicted in Figure 


4.5 Remarks 


In Sections |4.1| through |4.4[ we have drawn conclusions by surveying accepted concepts of 
vision and cognition. Notwithstanding, we state these conclusions as conjectures with theo¬ 
retical evidence and with examples. This is because the validation of these hypotheses would 
encompass vast experimentation; producing material enough to spam a few papers or a book, 
to be conservative. Therefore, we leave our discussions both as contributions - to guide new 
systematizations; and, as future work - to drive further rehnements and discoveries. 

Our conclusions also point to a challenging systematization. Rendering all the recommended 
aspects, together with multiple techniques and data domains, might involve a development ef¬ 
fort similar to that of huge software pieces, as office suites for instance. This is academically 
non-attractive, and economically risky. Possibly, the solution is to set a well-dehned devel¬ 
opment framework for collaborative work, with ample acceptation, and standard interfacing. 
In the realm of machine learning, software Weka ( |Hall et ah 2009) has achieved great success 
in a similar endeavor. However, the graphical nature of InfoVis, and its data preprocessing 
techniques, imposes big challenges. 


5 Conclusions 

We reviewed concepts on vision, cognition, and Information Visualization by introducing the 
organizational theorization named Visual Expression Process, which proposes a course of action 
to explain visual data analysis. Over an extensive literature survey, the theorization provides 
comprehension and science for data graphical presentation. It proposes a new perspective to 
discuss and characterize techniques. From this perspective, for instance, it would be possible 
to “dissect” a given technique in terms of the channels, the analytical perceptions and the 
patterns that it supports, potentially revealing strengths and weaknesses. Our contributions 
are as follows: 
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• Theoretical compendium; we plotted the Visual Expression Process to interrelate vision, 
cognition, and Information Visualization; 

• Discussions: we provided an extensive survey from different helds of science to serve as 
basis for relevant debate; 

• Reflections: we revisited design practices supported by examples and study cases. 

Overall, we have put together key concepts to introduce an insightful consideration of vi¬ 
sualizations. The reductionist perspective of our organization translates non-familiar concepts 
of vision and cognition into their corresponding effects with respect to design. In the form 
of a vocabulary taxonomically organized, we proposed a simplified comprehension of the fac¬ 
tors that dehne techniques and systems. With our contributions, we expect to foster a more 
comprehensive, accessible, and applied science of visualization. 
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